Method of generating a clipping from a printed substrate

ABSTRACT

A method of generating a clipping from a printed substrate. The substrate has a graphic image and a position-coding pattern disposed thereon and the position-coding pattern identifies an identity and a plurality of locations. The method comprises the steps of: (i) receiving, in a computer system, interaction data indicative of the identity and a position of a sensing device relative to the substrate, the sensing device generating the interaction data, when operatively positioned or moved relative to the substrate, by reading the position-coding pattern; (ii) identifying and retrieving a page description corresponding to the printed substrate using the identity, the page description including a visual description describing the graphic image and an input description describing interactive elements; (iii) determining, using the page description and the position of the sensing device, a clipping zone of the substrate indicated by the sensing device, the clipping zone containing a part of the graphic image; (iv) generating the clipping, the clipping comprising a clipping identity identifying the page description and the clipping zone; and (v) copying the clipping to a clipboard.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

The present application claims the right of priority under 35 U.S.C. §119(e) based on U.S. Provisional Patent Application No. 60/974,077,filed Sep. 21, 2007 which is incorporated by reference herein in itsentirety as if fully set forth herein.

FIELD OF INVENTION

The present invention relates generally to computing systems and, moreparticularly, to a method and system for enabling selection and use ofobjects in a computer system. It has been developed primarily to enableusers to receive useful information from a paper-based user interface.

CO-PENDING APPLICATIONS

The following applications have been filed by the Applicantsimultaneously with the present application:

NPT087US NPT088US NPT089US NPT090US NPT092US NPT093US NPT094US NPT095USNPT096US NPT097US NPS151US NPP090US NPP091US NPP092US NPP093US NPP094USNPP095US NPP096US NPP097US NPS149US NPS150US NPZ033US NPP099US NPP100USNPP101US NPP102US NPP103US NPP104US NPP105US NPP106US NPP107US NPP108US

The disclosures of these co-pending applications are incorporated hereinby reference. The above applications have been identified by theirfiling docket number, which will be substituted with the correspondingapplication number, once assigned.

CROSS REFERENCES

The following patents or patent applications filed by the applicant orassignee of the present invention are hereby incorporated bycross-reference.

10/815621 10/815635 10/815647 11/488162 10/815636 11/041652 11/04160911/041556 10/815609 7204941 7278727 10/913380 7122076 7156289 09/5751976720985 7295839 09/722174 7068382 7094910 7062651 6644642 65499356987573 6727996 6760119 7064851 6290349 6428155 6785016 6831682 67418716965439 10/932044 6870966 6474888 6724374 6788982 7263270 67882936737591 09/693514 10/778056 10/778061 11/193482 7055739 6830196 71822477082562 10/409864 7108192 10/492169 10/492152 10/492168 10/4921617308148 6957768 7170499 11856061 11/672522 11/672950 11754310 120155077148345 12025746 12025762 12025765 10/407212 6902255 6755509

BACKGROUND

The Applicant has previously described a method of enabling users toaccess information from a computer system via a printed substrate e.g.paper. The substrate has coded data printed thereon, which is read by anoptical sensing device when the user interacts with the substrate usingthe sensing device. A computer receives interaction data from thesensing device and uses this data to determine what action is beingrequested by the user. For example, a user may select a printedhyperlink using a sensing device and retrieve a corresponding webpagevia a display device or printer.

It would be desirable to enhance the functionality of theabove-described system. Enhanced functionality would encourage greateruse of the system and, hence, generate increased revenue streams forsystem providers.

More particularly, it would be desirable to provide users with theability to complement web-based interactions with paper-basedinteractions.

SUMMARY OF INVENTION

In a first aspect the present invention provides a method of generatinga clipping from a printed substrate, said substrate having a graphicimage and a position-coding pattern disposed thereon, saidposition-coding pattern identifying an identity and a plurality oflocations, said method comprising the steps of:

receiving, in a computer system, interaction data indicative of theidentity and at least one position of a sensing device relative to thesubstrate, the sensing device generating the interaction data, whenoperatively positioned or moved relative to the substrate, by reading atleast some of the position-coding pattern;

identifying and retrieving at least part of a page descriptioncorresponding to the printed substrate using the identity, said pagedescription including a visual description describing the graphic imageand an input description describing one or more interactive elements;

determining, using the page description and the at least one position ofthe sensing device, a clipping zone of the substrate indicated by thesensing device, said clipping zone containing a part of the graphicimage;

generating the clipping, said clipping comprising a clipping identityidentifying at least said page description and said clipping zone; and

copying the clipping to a clipboard.

Optionally, said clipping identifies a clipping image, said clippingimage comprising a clipping visual description describing the part ofthe graphic image contained in said clipping zone.

Optionally, the method further comprising the step of storing saidclipping image on a clipping image store, wherein said stored clippingimage is indexed with said clipping identity.

Optionally, the method further comprising displaying said clipping on adisplay device, said displaying comprising the steps of:

-   -   retrieving said clipping image from said clipping image store        using said clipping identity; and    -   displaying said clipping image on the display device.

Optionally, the method further comprising displaying said clipping on adisplay device, said displaying comprising the steps of:

-   -   identifying said page description using said clipping identity;    -   rendering a clipping image from said page description, said        clipping image comprising a clipping visual description        describing the part of the graphic image contained in said        clipping zone; and    -   displaying said clipping image on the display device.

Optionally, said clipping zone is indicated by a path of the sensingdevice relative to the substrate.

Optionally, said clipping identity identifies a time at which saidclipping is generated.

Optionally, said clipping zone contains at least one interactiveelement, such that said clipping is an interactive clipping.

Optionally, said at least one interactive element is selected from thegroup comprising: a hyperlink and an input field.

Optionally, said clipping identity identifies a clipping inputdescription describing said at least one interactive element via saidpage description and said clipping zone.

Optionally, said clipping further identifies a clipping input serviceconfigured for interpreting an interaction with said clipping via agraphical user interface.

Optionally, said clipping further identifies citation informationassociated with the part of the graphic image contained in said clippingzone.

Optionally, the clipping is represented, at least partially, using anHTML fragment containing an image map.

Optionally, the image map is selected from the group comprising: aserver-side image map and a client-side image map.

Optionally, said clipping is represented using a script.

Optionally, the method further comprising at least one of: pasting theclipping into an application, e-mailing the clipping, and pasting theclipping into a blog.

Optionally, the method further comprising the steps of:

-   -   displaying said clipping; and    -   interacting with said at least one interactive element using        GUI-based software supported by an operating system of a display        device.

Optionally, the interaction with the at least one interactive elementidentifies a click location and the clipping identity.

Optionally, the clipping identity and the click location are used toidentify the corresponding page description and interpret theinteraction.

Optionally, the method further comprising identifying a target URI anddisplaying a corresponding target resource on a display device.

In a second aspect the present invention provides a method of generatinga self-contained interactive clipping from a printed substrate, saidsubstrate having a graphic image and a position-coding pattern disposedthereon, said position-coding pattern identifying an identity and aplurality of locations, said method comprising the steps of:

-   -   receiving, in a computer system, interaction data indicative of        the identity and at least one position of a sensing device        relative to the substrate, the sensing device generating the        interaction data, when operatively positioned or moved relative        to the substrate, by reading at least some of the        position-coding pattern;    -   identifying and retrieving at least part of a page description        corresponding to the printed substrate using the identity, said        page description including a visual description describing the        graphic image and an input description describing one or more        interactive elements;    -   determining, using the page description and the at least one        position of the sensing device, a clipping zone of the substrate        indicated by the sensing device, said clipping zone containing a        part of the graphic image;    -   generating the clipping using said page description and said        clipping zone, said clipping comprising a clipping visual        description describing the part of the graphic image contained        in said clipping zone and a clipping input description        describing at least one interactive element contained in said        clipping zone; and    -   copying the clipping to a clipboard.

Optionally, the method further comprising displaying said clipping on adisplay device.

Optionally, said clipping zone is indicated by a path of the sensingdevice relative to the substrate.

Optionally, said at least one interactive element is selected from thegroup comprising: a hyperlink and an input field.

Optionally, the clipping is represented, at least partially, using anHTML fragment containing an image map explicitly hyperlinked to at leastone target URI.

Optionally, the image map is selected from the group comprising: aserver-side image map and a client-side image map.

Optionally, said clipping is represented using a script.

Optionally, the method further comprising at least one of: pasting theclipping into an application, e-mailing the clipping, and pasting theclipping into a blog.

Optionally, the method further comprising the steps of:

-   -   displaying said clipping on a display device; and    -   interacting with an interactive element using GUI-based software        supported by an operating system of the display device.

Optionally, the interaction with the interactive element identifies aclick location and a target URI using the clipping input descriptioncontained in the clipping.

Optionally, the method further comprising displaying a target resourcecorresponding to said target URI on said display device.

BRIEF DESCRIPTION OF DRAWINGS

Preferred and other embodiments of the invention will now be described,by way of non-limiting example only, with reference to the accompanyingdrawings, in which:

FIG. 1 is a schematic of the relationship between a sample printednetpage and its online page description;

FIG. 2 shows an embodiment of basic netpage architecture with variousalternatives for the relay device;

FIG. 3 is a schematic view of a high-level structure of a printednetpage and its online page description;

FIG. 4 is a schematic view of a document and page description classdiagram;

FIG. 5 is a schematic view of a document and page ownership classdiagram;

FIG. 6 is a schematic view of a terminal element specialization classdiagram;

FIG. 7 is a schematic view of a static element specialization classdiagram;

FIG. 8 illustrates a collection of netpage servers, Web terminals,printers and relays interconnected via a network;

FIG. 9 is a schematic view of a pen class diagram;

FIG. 10 is a perspective view of a netpage pen and its associatedtag-sensing field-of-view cone;

FIG. 11 is a perspective exploded view of the netpage pen shown in FIG.8;

FIG. 12 is a schematic block diagram of a pen controller for the netpagepen shown in FIGS. 10 and 11;

FIG. 13 is a schematic view of a hyperlink element class diagram;

FIG. 14 is a schematic view of a hyperlink element specialization classdiagram;

FIG. 15 is a schematic view of a hyperlinked group class diagram;

FIG. 16 is a schematic view of a form class diagram;

FIG. 17 is a schematic view of a digital ink class diagram;

FIG. 18 is a schematic view of a field element specialization classdiagram;

FIG. 19 is a schematic view of a checkbox field class diagram;

FIG. 20 is a schematic view of a text field class diagram;

FIG. 21 is a schematic view of a signature field class diagram;

FIG. 22 is a flowchart of an input processing algorithm;

FIG. 22A is a detailed flowchart of one step of the flowchart of FIG.22;

FIG. 23 is a schematic view of a hyperlink request class diagram;

FIG. 24 is a schematic view of an article group class diagram;

FIG. 25 is a schematic view of a selection hyperlink class diagram;

FIG. 26 is a schematic view of a selection page server command classdiagram;

FIG. 27 is a schematic view of a selection retrieval protocol;

FIG. 28 is a schematic view of a styled text object and formattedtextflow fragment class diagram;

FIG. 29 is a schematic view of a page fragment bearing some printedtext, with two words of the text selected by circumscription (lasso);

FIG. 30 is a schematic view of a page fragment bearing a set of commandsthat operate on the current selection;

FIG. 31 is a styled paragraph and formatted text line object diagram;

FIG. 32 is a printed netpage with a clipping stroke;

FIG. 33 is a virtual clipping;

FIG. 34 is an interactive clipping class diagram;

FIG. 35 is a clipping display and click-through interaction diagram;

FIG. 36 is an interactive clipping as shown in a graphical HTML client;and

FIG. 37 is an interactive clipping as shown in a non-graphical HTMLclient.

1NETPAGE SYSTEM OVERVIEW 1.1 Netpage System Architecture

The present invention is used in connection with the Applicant's netpagesystem, which has been described in detail in the cross-referencedpatent applications identified above.

In brief summary, the preferred form of the netpage system employs acomputer interface in the form of a mapped surface, that is, a physicalsurface which contains references to a map of the surface maintained ina computer system. The map references can be queried by an appropriatesensing device. Depending upon the specific implementation, the mapreferences may be encoded visibly or invisibly, and defined in such away that a local query on the mapped surface yields an unambiguous mapreference both within the map and among different maps. The computersystem can contain information about features on the mapped surface, andsuch information can be retrieved based on map references supplied by asensing device used with the mapped surface. The information thusretrieved can take the form of actions which are initiated by thecomputer system on behalf of the operator in response to the operator'sinteraction with the surface features.

In its preferred form, the netpage system relies on the production of,and human interaction with, netpages. These are pages of text, graphicsand images printed on ordinary paper, but which work like interactivewebpages. Information is encoded on each page using ink which issubstantially invisible to the unaided human eye. The ink, however, andthereby the coded data, can be sensed by an optically imaging sensingdevice (“reader”) and transmitted to the netpage system. The sensingdevice may take the form of, for example, a clicker (for clicking on aspecific position on a surface), a pointer having a stylus (for pointingor gesturing on a surface using pointer strokes), or a pen having amarking nib (for marking a surface with ink when pointing, gesturing orwriting on the surface). Any references herein to “pen” or “netpage pen”are provided by way of example only. It will, of course, be appreciatedthat the netpage pen may take the form of any suitable optically imagingsensing device or reader.

In one embodiment, active buttons and hyperlinks on each page can beclicked with the sensing device to request information from the networkor to signal preferences to a network server. In one embodiment, textwritten by hand on a netpage is automatically recognized and convertedto computer text in the netpage system, allowing forms to be filled in.In other embodiments, signatures recorded on a netpage are automaticallyverified, allowing e-commerce transactions to be securely authorized. Inother embodiments, text on a netpage may be clicked or gestured toinitiate a search based on keywords indicated by the user.

As illustrated in FIG. 1, a printed netpage 1 can represent aninteractive form which can be filled in by the user both physically, onthe printed page, and “electronically”, via communication between thepen and the netpage system. The example shows a “Request” formcontaining name and address fields and a submit button. The netpage 1consists of a graphic impression 2, printed using visible ink, and asurface coding pattern 3 superimposed with the graphic impression. Thecoding pattern 3 is typically printed with an infrared ink and thesuperimposed graphic impression 2 is printed with colored ink(s) havinga complementary infrared window, allowing infrared imaging of the codingpattern 3.

The surface coding pattern 3 typically takes the form of a grid oftarget dots, which comprises a collection of tags 4. One such tag 4 isshown in the shaded region of FIG. 1, although it will be appreciatedthat contiguous tags 4, defined by the coding pattern 3, are denselytiled over the whole netpage 1.

A corresponding page description 5, stored on the netpage network,describes the individual elements of the netpage. In particular it hasan input description describing the type and spatial extent (zone) ofeach interactive element (i.e. text field or button in the example), toallow the netpage system to correctly interpret input via the netpage.The submit button 6, for example, has a zone 7 which corresponds to thespatial extent of the corresponding graphic 8.

As illustrated in FIG. 2, a netpage reader 400 (e.g. netpage pen) worksin conjunction with a netpage relay device 601, which is anInternet-connected device for home, office or mobile use. The reader 400is wireless and communicates securely with the netpage relay device 601via a short-range radio link 9. In an alternative embodiment, the reader400 utilizes a wired connection, such as a USB or other serialconnection, to the relay device 601.

The relay device 601 performs the basic function of relaying interactiondata to a page server 10, which interprets the interaction data. Asshown in FIG. 2, the relay device 601 may, for example, take the form ofa personal computer 601 a, a netpage printer 601 b or some other relay601 c (e.g. personal computer or mobile phone incorporating a webbrowser).

The netpage printer 601 b is able to deliver, periodically or on demand,personalized newspapers, magazines, catalogs, brochures and otherpublications, all printed at high quality as interactive netpages.Unlike a personal computer, the netpage printer is an appliance whichcan be, for example, wall-mounted adjacent to an area where the morningnews is first consumed, such as in a user's kitchen, near a breakfasttable, or near the household's point of departure for the day. It alsocomes in tabletop, desktop, portable and miniature versions. Netpagesprinted on-demand at their point of consumption combine the ease-of-useof paper with the timeliness and interactivity of an interactive medium.

Alternatively, the netpage relay device 601 may be a portable device,such as a mobile phone or PDA, a laptop or desktop computer, or aninformation appliance connected to a shared display, such as a TV. Ifthe relay device 601 is not a netpage printer 601 b which printsnetpages digitally and on demand, the netpages may be printed bytraditional analog printing presses, using such techniques as offsetlithography, flexography, screen printing, relief printing androtogravure, as well as by digital printing presses, using techniquessuch as drop-on-demand inkjet, continuous inkjet, dye transfer, andlaser printing. As will be explained in more detail below, the presentinvention provides an alternative means which enables users convenientlyto generate netpages, as well as scan or print images.

As shown in FIG. 2, the netpage reader 400 interacts with a portion ofthe position-coding tag pattern on a printed netpage 1, or other printedsubstrate such as a label of a product item 251, and communicates, via ashort-range radio link 9, the interaction to the relay device 601. Therelay 601 sends corresponding interaction data to the relevant netpagepage server 10 for interpretation. Raw data received from the netpagereader 400 may be relayed directly to the page server 10 as interactiondata. Alternatively, the interaction data may be encoded in the form ofan interaction URI and transmitted to the page server 10 via a user'sweb browser 601 c. The web browser 601 c may then receive a URI from thepage server 10 and access a webpage via a webserver 201. In somecircumstances, the page server 10 may access application computersoftware running on a netpage application server 13.

The netpage relay device 601 can be configured to support any number ofreaders 400, and a reader can work with any number of netpage relays. Inthe preferred implementation, each netpage reader 400 has a uniqueidentifier. This allows each user to maintain a distinct profile withrespect to a netpage page server 10 or application server 13.

A netpage pen may be registered with a netpage registration server 11and linked to one or more payment card accounts. This allows e-commercepayments to be securely authorized using the netpage pen. The netpageregistration server compares the signature captured by the netpage penwith a previously registered signature, allowing it to authenticate theuser's identity to an e-commerce server. Other biometrics can also beused to verify identity. One version of the netpage pen includesfingerprint scanning, verified in a similar way by the netpageregistration server.

1.2 Netpages

Netpages are the foundation on which a netpage network is built. Theyprovide a paper-based user interface to published information andinteractive services.

As shown in FIG. 1, a netpage consists of a printed page (or othersurface region) invisibly tagged with references to an onlinedescription 5 of the page. The online page description 5 is maintainedpersistently by the netpage page server 10. The page description has avisual description describing the visible layout and content of thepage, including text, graphics and images. It also has an inputdescription describing the input elements on the page, includingbuttons, hyperlinks, and input fields. A netpage allows markings madewith a netpage pen on its surface to be simultaneously captured andprocessed by the netpage system.

Multiple netpages (for example, those printed by analog printingpresses) can share the same page description. However, to allow inputthrough otherwise identical pages to be distinguished, each netpage maybe assigned a unique page identifier in the form of a page ID (or, moregenerally, an impression ID). The page ID has sufficient precision todistinguish between a very large number of netpages.

Each reference to the page description 5 is repeatedly encoded in thenetpage pattern. Each tag (and/or a collection of contiguous tags)identifies the unique page on which it appears, and thereby indirectlyidentifies the page description 5. Each tag also identifies its ownposition on the page. Characteristics of the tags are described in moredetail below.

Tags are typically printed in infrared-absorptive ink on any substratewhich is infrared-reflective, such as ordinary paper, or in infraredfluorescing ink. Near-infrared wavelengths are invisible to the humaneye but are easily sensed by a solid-state image sensor with anappropriate filter.

A tag is sensed by a 2D area image sensor in the netpage reader 400, andthe tag data is transmitted to the netpage system via the nearestnetpage relay device 601. The reader 400 is wireless and communicateswith the netpage relay device 601 via a short-range radio link. It isimportant that the reader recognize the page ID and position on everyinteraction with the page, since the interaction is stateless. Tags areerror-correctably encoded to make them partially tolerant to surfacedamage.

The netpage page server 10 maintains a unique page instance for eachunique printed netpage, allowing it to maintain a distinct set ofuser-supplied values for input fields in the page description 5 for eachprinted netpage 1.

2 NETPAGE TAGS

Each tag 4, contained in the position-coding pattern 3, identifies anabsolute location of that tag within a region of a substrate.

Each interaction with a netpage should also provide a region identitytogether with the tag location. In a preferred embodiment, the region towhich a tag refers coincides with an entire page, and the region ID istherefore synonymous with the page ID of the page on which the tagappears. In other embodiments, the region to which a tag refers can bean arbitrary subregion of a page or other surface. For example, it cancoincide with the zone of an interactive element, in which case theregion ID can directly identify the interactive element.

As described in some of the Applicant's previous applications (e.g. U.S.Pat. No. 6,832,717), the region identity may be encoded discretely ineach tag 4. As described other of the Applicant's applications (e.g.U.S. application Ser. Nos. 12/025,746 & 12/025,765 filed on Feb. 5,2008), the region identity may be encoded by a plurality of contiguoustags in such a way that every interaction with the substrate stillidentifies the region identity, even if a whole tag is not in the fieldof view of the sensing device.

Each tag 4 should preferably identify an orientation of the tag relativeto the substrate on which the tag is printed. Strictly speaking, eachtag 4 identifies an orientation of tag data relative to a gridcontaining the tag data. However, since the grid is typically orientedin alignment with the substrate, then orientation data read from a tagenables the rotation (yaw) of the netpage reader 400 relative to thegrid, and thereby the substrate, to be determined.

A tag 4 may also encode one or more flags which relate to the region asa whole or to an individual tag. One or more flag bits may, for example,signal a netpage reader 400 to provide feedback indicative of a functionassociated with the immediate area of the tag, without the reader havingto refer to a corresponding page description 5 for the region. A netpagereader may, for example, illuminate an “active area” LED when positionedin the zone of a hyperlink.

A tag 4 may also encode a digital signature or a fragment thereof. Tagsencoding digital signatures (or a part thereof) are useful inapplications where it is required to verify a product's authenticity.Such applications are described in, for example, US Publication No.2007/0108285, the contents of which is herein incorporated by reference.The digital signature may be encoded in such a way that it can beretrieved from every interaction with the substrate. Alternatively, thedigital signature may be encoded in such a way that it can be assembledfrom a random or partial scan of the substrate.

It will, of course, be appreciated that other types of information (e.g.tag size etc) may also be encoded into each tag or a plurality of tags.

For a full description of netpage tags 4, reference is made to theApplicant's previously filed patent applications identified above, thecontents of which are herein incorporated by reference (see, forexample, U.S. Pat. No. 6,832,717 and U.S. application Ser. Nos.12/025,746 & 12/025,765 filed on Feb. 5, 2008, the contents of which areherein incorporated by reference).

3 DOCUMENT AND PAGE DESCRIPTIONS

A preferred embodiment of a document and page description class diagramis shown in FIGS. 4 and 5.

In the netpage system a document is described at three levels. At themost abstract level the document 836 has a hierarchical structure whoseterminal elements 839 are associated with content objects 840 such astext objects, text style objects, image objects, etc. Once the documentis printed on a printer with a particular page size, the document ispaginated and otherwise formatted. Formatted terminal elements 835 willin some cases be associated with content objects which are differentfrom those associated with their corresponding terminal elements,particularly where the content objects are style-related. Each printedinstance of a document and page is also described separately, to allowinput captured through a particular page instance 830 to be recordedseparately from input captured through other instances of the same pagedescription.

The presence of the most abstract document description on the pageserver allows a a copy of a document to be printed without being forcedto accept the source document's specific format. The user or a printingpress may be requesting a copy for a printer with a different page size,for example. Conversely, the presence of the formatted documentdescription on the page server allows the page server to efficientlyinterpret user actions on a particular printed page.

A formatted document 834 consists of a set of formatted pagedescriptions 5, each of which consists of a set of formatted terminalelements 835. Each formatted element has a spatial extent or zone 58 onthe page. This defines the active area of input elements such ashyperlinks and input fields.

A document instance 831 corresponds to a formatted document 834. Itconsists of a set of page instances 830, each of which corresponds to apage description 5 of the formatted document. Each page instance 830describes a single unique printed netpage 1, and records the page ID 50of the netpage. A page instance is not part of a document instance if itrepresents a copy of a page requested in isolation.

A page instance consists of a set of terminal element instances 832. Anelement instance only exists if it records instance-specificinformation. Thus, a hyperlink instance exists for a hyperlink elementbecause it records a transaction ID 55 which is specific to the pageinstance, and a field instance exists for a field element because itrecords input specific to the page instance. An element instance doesnot exist, however, for static elements such as textflows.

A terminal element 839 can be a visual element or an input element. Avisual element can be a static element 843 or a dynamic element 846. Aninput element can be a hyperlink element 844 or a field element 845, asshown in FIG. 6. A static element 843 can be a style element 847 with anassociated style object 854, a textflow element 848 with an associatedstyled text object 855, an image element 849 with an associated imageelement 856, a graphic element 850 with an associated graphic object857, a video clip element 851 with an associated video clip object 858,an audio clip element 852 with an associated audio clip object 859, or ascript element 853 with an associated script object 860, as shown inFIG. 7.

A page instance has a background field 833 which is used to record anydigital ink captured on the page which does not apply to a specificinput element.

In the preferred form of the invention, a tag map 811 is associated witheach page instance to allow tags on the page to be translated intolocations on the page.

4 THE NETPAGE NETWORK

A simple netpage network may consist of a distributed set of netpagepage servers 10, netpage registration servers 11, netpage ID servers 12,netpage application servers 13, and netpage relay devices 601 connectedvia a network 19 such as the Internet, as shown in FIG. 8. Of course,other services may be incorporated into the Netpage network, such asthose enabling an interactive clipping service as shown in FIG. 35.

The netpage registration server 11 is a server which recordsrelationships between users, pens, printers and/or applications, andthereby authorizes various network activities. It authenticates usersand acts as a signing proxy on behalf of authenticated users inapplication transactions. It also provides handwriting recognitionservices. As described above, a netpage page server 10 maintainspersistent information about page descriptions and page instances. Thenetpage network includes any number of page servers, each handling asubset of page instances. Since a page server also maintains user inputvalues for each page instance, clients such as netpage relays 601 sendnetpage input directly to the appropriate page server. The page serverinterprets any such input relative to the description of thecorresponding page.

A netpage ID server 12 allocates document IDs 51 on demand, and providesload-balancing of page servers via its ID allocation scheme.

A netpage relay 601 uses the Internet Distributed Name System (DNS), orsimilar, to resolve a netpage page ID 50 into the network address of thenetpage page server handling the corresponding page instance.

A netpage application server 13 is a server which hosts interactivenetpage applications.

Netpage servers can be hosted on a variety of network server platformsfrom manufacturers such as IBM, Hewlett-Packard, and Sun. Multiplenetpage servers can run concurrently on a single host, and a singleserver can be distributed over a number of hosts. Some or all of thefunctionality provided by netpage servers, and in particular thefunctionality provided by the ID server and the page server, can also beprovided directly in a netpage appliance such as a netpage printer, in acomputer workstation, or on a local network.

5 THE NETPAGE PEN

The active sensing device (or reader 400) of the netpage system may takethe form of, for example, a clicker (for clicking on a specific positionon a surface), a pointer having a stylus (for pointing or gesturing on asurface using pointer strokes), or a pen having a marking nib (formarking a surface with ink when pointing, gesturing or writing on thesurface). A pen 101 is described herein, although it will be appreciatedthat clickers and pointers may be of similar construction. The pen 101uses its embedded controller 134 to capture and decode netpage tags froma page via an image sensor. The image sensor is a solid-state deviceprovided with an appropriate filter to permit sensing at onlynear-infrared wavelengths. As described in more detail below, the systemis able to sense when the nib is in contact with the surface, and thepen is able to sense tags at a sufficient rate to capture humanhandwriting (i.e. at 200 dpi or greater and 100 Hz or faster).Information captured by the pen may be encrypted and wirelesslytransmitted to the printer (or base station), the printer or basestation interpreting the data with respect to the (known) pagestructure.

One embodiment of the netpage pen 101 operates both as a normal markingink pen and as a non-marking stylus (i.e. as a pointer). The markingaspect, however, is not necessary for using the netpage system as abrowsing system, such as when it is used as an Internet interface. Eachnetpage pen is registered with the netpage system and has a unique penID 61. FIG. 11 shows the netpage pen class diagram, reflectingpen-related information maintained by a registration server 11 on thenetpage network.

When the nib is in contact with a netpage, the pen determines itsposition and orientation relative to the page. The nib is attached to aforce sensor, and the force on the nib is interpreted relative to athreshold to indicate whether the pen is “up” or “down”. This allows ainteractive element on the page to be ‘clicked’ by pressing with the pennib, in order to request, say, information from a network. Furthermore,the force is captured as a continuous value to allow, say, the fulldynamics of a signature to be verified.

The pen determines the position and orientation of its nib on thenetpage by imaging, in the infrared spectrum, an area 193 of the page inthe vicinity of the nib. It decodes the nearest tag and computes theposition of the nib relative to the tag from the observed perspectivedistortion on the imaged tag and the known geometry of the pen opticsi.e. the spatial relationship between the nib and the optical axis.Although the position resolution of the tag may be low, because the tagdensity on the page is inversely proportional to the tag size, theadjusted position resolution is quite high, exceeding the minimumresolution required for accurate handwriting recognition.

Pen actions relative to a netpage are captured as a series of strokes. Astroke consists of a sequence of time-stamped pen positions on the page,initiated by a pen-down event and completed by the subsequent pen-upevent. A stroke is also tagged with the page ID 50 of the netpagewhenever the page ID changes, which, under normal circumstances, is atthe commencement of the stroke.

Each netpage pen has a current selection 826 associated with it,allowing the user to perform copy and paste operations etc. Theselection is timestamped to allow the system to discard it after adefined time period. The current selection describes a region of a pageinstance. It consists of the most recent digital ink stroke capturedthrough the pen relative to the background area of the page. It isinterpreted in an application-specific manner once it is submitted to anapplication via a selection hyperlink activation.

Each pen has a current nib 824. This is the nib last notified by the pento the system. In the case of the default netpage pen described above,either the marking black ink nib or the non-marking stylus nib iscurrent. Each pen also has a current nib style 825. This is the nibstyle last associated with the pen by an application, e.g. in responseto the user selecting a color from a palette. The default nib style isthe nib style associated with the current nib. Strokes captured througha pen are tagged with the current nib style. When the strokes aresubsequently reproduced, they are reproduced in the nib style with whichthey are tagged.

The pen 101 may have one or more buttons 209, which are pressed by theuser to select a mode of the pen. The button(s) are used to determine abehavior of the pen, which, in turn, determines how a stroke isinterpreted by the page server 10. For example, a ‘clipping mode’ may beselected using a button 209 (see Section 7.3).

Whenever the pen is within range of a relay device 601 with which it cancommunicate, the pen slowly flashes its “online” LED. When the pen failsto decode a stroke relative to the page, it momentarily activates its“error” LED. When the pen succeeds in decoding a stroke relative to thepage, it momentarily activates its “ok” LED.

A sequence of captured strokes is referred to as digital ink. Digitalink forms the basis for the digital exchange of drawings andhandwriting, for online recognition of handwriting, and for onlineverification of signatures.

The pen is wireless and transmits digital ink to the relay device 601via a short-range radio link. The transmitted digital ink is encryptedfor privacy and security and packetized for efficient transmission, butis always flushed on a pen-up event to ensure timely handling in theprinter.

When the pen is out-of-range of a relay device 601 it buffers digitalink in internal memory, which has a capacity of over ten minutes ofcontinuous handwriting. When the pen is once again within range of arelay device, it transfers any buffered digital ink.

A pen can be registered with any number of relay devices, but becauseall state data resides in netpages both on paper and on the network, itis largely immaterial which relay device a pen is communicating with atany particular time.

A more detailed description of the Netpage pen may be found in theApplicant's earlier cross-referenced patents and patent applications(see, for example, U.S. Pat. No. 7,105,753; U.S. Pat. No. 7,015,901;U.S. Pat. No. 7,091,960; and US Publication No. 2006/0028459, thecontents of which are herein incorporated by reference).

5.1 Pen Mechanics

Referring to FIGS. 10 and 11, the pen, generally designated by referencenumeral 101, includes a housing 102 in the form of a plastics mouldinghaving walls 103 defining an interior space 104 for mounting the pencomponents. Mode selector buttons 209 are provided on the housing 102.The pen top 105 is in operation rotatably mounted at one end 106 of thehousing 102. A semi-transparent cover 107 is secured to the opposite end108 of the housing 102. The cover 107 is also of moulded plastics, andis formed from semi-transparent material in order to enable the user toview the status of the LED mounted within the housing 102. The cover 107includes a main part 109 which substantially surrounds the end 108 ofthe housing 102 and a projecting portion 110 which projects back fromthe main part 109 and fits within a corresponding slot 111 formed in thewalls 103 of the housing 102. A radio antenna 112 is mounted behind theprojecting portion 110, within the housing 102. Screw threads 113surrounding an aperture 113 A on the cover 107 are arranged to receive ametal end piece 114, including corresponding screw threads 115. Themetal end piece 114 is removable to enable ink cartridge replacement.

Also mounted within the cover 107 is a tri-color status LED 116 on aflex PCB 117. The antenna 112 is also mounted on the flex PCB 117. Thestatus LED 116 is mounted at the top of the pen 101 for good all-aroundvisibility.

The pen described herein can operate both as a normal marking ink penand as a non-marking stylus (‘pointer’). An ink pen cartridge 118 withnib 119 and a stylus 120 with stylus nib 121 are mounted side by sidewithin the housing 102. Either the ink cartridge nib 119 or the stylusnib 121 can be brought forward through open end 122 of the metal endpiece 114, by rotation of the pen top 105. Respective slider blocks 123and 124 are mounted to the ink cartridge 118 and stylus 120,respectively. A rotatable cam barrel 125 is secured to the pen top 105in operation and arranged to rotate therewith. The cam barrel 125includes a cam 126 in the form of a slot within the walls 181 of the cambarrel. Cam followers 127 and 128 projecting from slider blocks 123 and124 fit within the cam slot 126. On rotation of the cam barrel 125, theslider blocks 123 or 124 move relative to each other to project eitherthe pen nib 119 or stylus nib 121 out through the hole 122 in the metalend piece 114. The pen 101 has three states of operation. By turning thetop 105 through 90 ° steps, the three states are:

-   Stylus 120 nib 121 out;-   Ink cartridge 118 nib 119 out; and-   Neither ink cartridge 118 nib 119 out nor stylus 120 nib 121 out.

A second flex PCB 129, is mounted on an electronics chassis 130 whichsits within the housing 102. The second flex PCB 129 mounts an infraredLED 131 for providing infrared radiation for projection onto thesurface. An image sensor 132 is provided mounted on the second flex PCB129 for receiving reflected radiation from the surface. The second flexPCB 129 also mounts a radio frequency chip 133, which includes an RFtransmitter and RF receiver, and a controller chip 134 for controllingoperation of the pen 101. An optics block 135 (formed from moulded clearplastics) sits within the cover 107 and projects an infrared beam ontothe surface and receives images onto the image sensor 132. Power supplywires 136 connect the components on the second flex PCB 129 to batterycontacts 137 which are mounted within the cam barrel 125. A terminal 138connects to the battery contacts 137 and the cam barrel 125. A threevolt rechargeable battery 139 sits within the cam barrel 125 in contactwith the battery contacts. An induction charging coil 140 is mountedabout the second flex PCB 129 to enable recharging of recharging of thebattery 139 via induction. The second flex PCB 129 also mounts aninfrared LED 143 and infrared photodiode 144 for detecting displacementin the cam barrel 125 when either the stylus 120 or the ink cartridge118 is used for writing, in order to enable a determination of the forcebeing applied to the surface by the pen nib 119 or stylus nib 121. TheIR photodiode 144 detects light from the IR LED 143 via reflectors (notshown) mounted on the slider blocks 123 and 124.

Rubber grip pads 141 and 142 are provided towards the end 108 of thehousing 102 to assist gripping the pen 101, and top 105 also includes aclip 142 for clipping the pen 101 to a pocket.

5.2 Pen Controller

The pen 101 is arranged to determine the position of its nib (stylus nib121 or ink cartridge nib 119 ) by imaging, in the infrared spectrum, anarea of the surface in the vicinity of the nib. It records the locationdata from the nearest location tag, and is arranged to calculate thedistance of the nib 121 or 119 from the location tab utilising optics135 and controller chip 134. The controller chip 134 calculates theorientation of the pen and the nib-to-tag distance from the perspectivedistortion observed on the imaged tag.

Utilising the RF chip 133 and antenna 112 the pen 101 can transmit thedigital ink data (which is encrypted for security and packaged forefficient transmission) to the computing system.

When the pen is in range of a relay device 601, the digital ink data istransmitted as it is formed. When the pen 101 moves out of range,digital ink data is buffered within the pen 101 (the pen 101 circuitryincludes a buffer arranged to store digital ink data for approximately12 minutes of the pen motion on the surface) and can be transmittedlater.

The nib ID allows the computing system to identify which nib (stylus nib121 or ink cartridge nib 119 ) is presently being used. The computingsystem can vary its operation depending upon which nib is being used.For example, if the ink cartridge nib 119 is being used the computingsystem may defer producing feedback output because immediate feedback isprovided by the ink markings made on the surface. Where the stylus nib121 is being used, the computing system may produce immediate feedbackoutput.

Since a user may change the nib 119, 121 between one stroke and thenext, the pen 101 optionally records a nib ID for a stroke 175. Thisbecomes the nib ID implicitly associated with later strokes.

Cartridges having particular nib characteristics may be interchangeablein the pen. The pen controller 134 may interrogate a cartridge to obtainthe nib ID 175 of the cartridge. The nib ID 175 may be stored in a ROMor a barcode on the cartridge. The controller 134 notifies the system ofthe nib ID whenever it changes. The system is thereby able to determinethe characteristics of the nib used to produce a stroke 175, and isthereby subsequently able to reproduce the characteristics of the strokeitself.

The controller chip 134 is mounted on the second flex PCB 129 in the pen101. FIG. 10 is a block diagram illustrating in more detail thearchitecture of the controller chip 134. FIG. 12 also showsrepresentations of the RF chip 133, the image sensor 132, the tri-colorstatus LED 116, the IR illumination LED 131, the IR force sensor LED143, and the force sensor photodiode 144.

Referring to FIG. 12, the pen controller chip 134 includes a controllingprocessor 145. Bus 146 enables the exchange of data between componentsof the controller chip 134. Flash memory 147 and a 512 KB DRAM 148 arealso included. An analog-to-digital converter 149 is arranged to convertthe analog signal from the force sensor photodiode 144 to a digitalsignal.

An image sensor interface 152 interfaces with the image sensor 132. Atransceiver controller 153 and base band circuit 154 are also includedto interface with the RF chip 133 which includes an RF circuit 155 andRF resonators and inductors 156 connected to the antenna 112.

The controlling processor 145 captures and decodes location data fromtags from the surface via the image sensor 132, monitors the forcesensor photodiode 144, controls the LEDs 116, 131 and 143, and handlesshort-range radio communication via the radio transceiver 153. It is amedium-performance (˜40 MHz) general-purpose RISC processor. Thecontrolling processor 145 also receives input via mode selectionbutton(s) 209, which can be used to control interpretation ofpen-substrate interactions.

The processor 145, digital transceiver components (transceivercontroller 153 and baseband circuit 154), image sensor interface 152,flash memory 147 and 512 KB DRAM 148 are integrated in a singlecontroller ASIC. Analog RF components (RF circuit 155 and RF resonatorsand inductors 156) are provided in the separate RF chip.

The image sensor 132 may be a 215×215 pixel CCD (such a sensor isproduced by Matsushita Electronic Corporation, and is described in apaper by Itakura, K T Nobusada, N Okusenya, R Nagayoshi, and M Ozaki, “A1 mm 50 k-Pixel IT CCD Image Sensor for Miniature Camera System”, IEEETransactions on Electronic Devices, Volt 47, number 1, January 2000,which is incorporated herein by reference) with an IR filter.

The controller ASIC 134 enters a quiescent state after a period ofinactivity when the pen 101 is not in contact with a surface. Itincorporates a dedicated circuit 150 which monitors the force sensorphotodiode 144 and wakes up the controller 134 via the power manager 151on a pen-down event.

The radio transceiver communicates in the unlicensed 900 MHz bandnormally used by cordless telephones, or alternatively in the unlicensed2.4 GHz industrial, scientific and medical (ISM) band, and usesfrequency hopping and collision detection to provide interference-freecommunication.

In an alternative embodiment, the pen incorporates an Infrared DataAssociation (IrDA) interface for short-range communication with a basestation or netpage printer.

In a further embodiment, the pen 101 includes a pair of orthogonalaccelerometers mounted in the normal plane of the pen 101 axis. Theaccelerometers 190 are shown in FIGS. 11 and 12 in ghost outline.

The provision of the accelerometers enables this embodiment of the pen101 to sense motion without reference to surface location tags, allowingthe location tags to be sampled at a lower rate. Each location tag IDcan then identify an object of interest rather than a position on thesurface. For example, if the object is a user interface input element(e.g. a command button), then the tag ID of each location tag within thearea of the input element can directly identify the input element.

The acceleration measured by the accelerometers in each of the x and ydirections is integrated with respect to time to produce aninstantaneous velocity and position.

Since the starting position of the stroke is not known, only relativepositions within a stroke are calculated. Although position integrationaccumulates errors in the sensed acceleration, accelerometers typicallyhave high resolution, and the time duration of a stroke, over whicherrors accumulate, is short.

6 NETPAGE INTERACTION

The netpage relay device 601 receives data relating to a stroke from thepen 101 when the pen is used to interact with a netpage 1. The codeddata 3 of the tags 4 is read by the pen when it is used to execute amovement, such as a stroke. The data allows the identity of theparticular page to be determined and an indication of the positioning ofthe pen relative to the page to be obtained. Interaction data comprisingthe page ID 50 and at least one position of the pen, is transmitted tothe relay device 601, where it resolves, via the DNS, the page ID 50 ofthe stroke into the network address of the netpage page server 10 whichmaintains the corresponding page instance 830. In one form of thenetpage system, the relay device 601 then transmits the stroke to thepage server. If the page was recently identified in an earlier stroke,then the relay device may already have the address of the relevant pageserver in its cache. Each netpage consists of a compact page layoutmaintained persistently by a netpage page server (see below). The pagelayout refers to objects such as images, fonts and pieces of text,typically stored elsewhere on the netpage network.

When the page server 10 receives the stroke from the pen, it retrievesthe page description to which the stroke applies, and determines whichelement of the page description the stroke intersects. It is then ableto interpret the stroke in the context of the type of the relevantelement.

A “click” is a stroke where the distance and time between the pen downposition and the subsequent pen up position are both less than somesmall maximum. An object which is activated by a click typicallyrequires a click to be activated, and accordingly, a longer stroke isignored. The failure of a pen action, such as a “sloppy” click, toregister may be indicated by the lack of response from the pen's “ok”LED.

There are two kinds of input elements in a netpage page description:hyperlinks and form fields. Input through a form field can also triggerthe activation of an associated hyperlink.

6.1 Hyperlinks

A hyperlink is a means of sending a message to a remote application, andtypically elicits a displayed or printed response in the netpage system.

A hyperlink element 844 identifies the application 71 which handlesactivation of the hyperlink, a link ID 54 which identifies the hyperlinkto the application, an “alias required” flag which asks the system toinclude the user's application alias ID 65 in the hyperlink activation,and a description which is used when the hyperlink is recorded as afavorite or appears in the user's history. The hyperlink element classdiagram is shown in FIG. 13.

When a hyperlink is activated, the page server sends a request to anapplication somewhere on the network. The application is identified byan application ID 64, and the application ID is resolved in the normalway via the DNS. There are three types of hyperlinks: general hyperlinks863, form hyperlinks 865, and selection hyperlinks 864, as shown in FIG.14. A general hyperlink can implement a request for a linked document,or may simply signal a preference to a server. A form hyperlink submitsthe corresponding form to the application. A selection hyperlink submitsthe current selection to the application. If the current selectioncontains a single-word piece of text, for example, the application mayreturn a single-page document giving the word's meaning within thecontext in which it appears, or a translation into a different language.Each hyperlink type is characterized by what information is submitted tothe application.

The corresponding hyperlink instance 862 records a transaction ID 55which can be specific to the page instance on which the hyperlinkinstance appears. The transaction ID can identify user-specific data tothe application, for example a “shopping cart” of pending purchasesmaintained by a purchasing application on behalf of the user.

The system includes the pen's current selection 826 in a selectionhyperlink activation. The system includes the content of the associatedform instance 868 in a form hyperlink activation, although if thehyperlink has its “submit delta” attribute set, only input since thelast form submission is included. The system includes an effectivereturn path in all hyperlink activations.

A hyperlinked group 866 is a group element 838 which has an associatedhyperlink, as shown in FIG. 15. When input occurs through any fieldelement in the group, the hyperlink 844 associated with the group isactivated. A hyperlinked group can be used to associate hyperlinkbehavior with a field such as a checkbox. It can also be used, inconjunction with the “submit delta” attribute of a form hyperlink, toprovide continuous input to an application. It can therefore be used tosupport a “blackboard” interaction model, i.e. where input is capturedand therefore shared as soon as it occurs.

6.2 Forms

A form defines a collection of related input fields used to capture arelated set of inputs through a printed netpage. A form allows a user tosubmit one or more parameters to an application software program runningon a server.

A form 867 is a group element 838 in the document hierarchy. Itultimately contains a set of terminal field elements 839. A forminstance 868 represents a printed instance of a form. It consists of aset of field instances 870 which correspond to the field elements 845 ofthe form. Each field instance has an associated value 871, whose typedepends on the type of the corresponding field element. Each field valuerecords input through a particular printed form instance, i.e. throughone or more printed netpages. The form class diagram is shown in FIG.16.

Each form instance has a status 872 which indicates whether the form isactive, frozen, submitted, void or expired. A form is active when firstprinted. A form becomes frozen once it is signed or once its freeze timeis reached. A form becomes submitted once one of its submissionhyperlinks has been activated, unless the hyperlink has its “submitdelta” attribute set. A form becomes void when the user invokes a voidform, reset form or duplicate form page command. A form expires when itsspecified expiry time is reached, i.e. when the time the form has beenactive exceeds the form's specified lifetime. While the form is active,form input is allowed. Input through a form which is not active isinstead captured in the background field 833 of the relevant pageinstance. When the form is active or frozen, form submission is allowed.Any attempt to submit a form when the form is not active or frozen isrejected, and instead elicits an form status report.

Each form instance is associated (at 59 ) with any form instancesderived from it, thus providing a version history. This allows all butthe latest version of a form in a particular time period to be excludedfrom a search.

All input is captured as digital ink. Digital ink 873 consists of a setof timestamped stroke groups 874, each of which consists of a set ofstyled strokes 875. Each stroke consists of a set of timestamped penpositions 876, each of which also includes pen orientation and nibforce. The digital ink class diagram is shown in FIG. 17.

A field element 845 can be a checkbox field 877, a text field 878, adrawing field 879, or a signature field 880. The field element classdiagram is shown in FIG. 18. Any digital ink captured in a field's zone58 is assigned to the field.

A checkbox field has an associated boolean value 881, as shown in FIG.19. Any mark (a tick, a cross, a stroke, a fill zigzag, etc.) capturedin a checkbox field's zone causes a true value to be assigned to thefield's value.

A text field has an associated text value 882, as shown in FIG. 20. Anydigital ink captured in a text field's zone is automatically convertedto text via online handwriting recognition, and the text is assigned tothe field's value. Online handwriting recognition is well-understood(see, for example, Tappert, C., C. Y. Suen and T. Wakahara, “The Stateof the Art in On-Line Handwriting Recognition”, IEEE Transactions onPattern Analysis and Machine Intelligence, Vol. 12, No. 8, August 1990,the contents of which are herein incorporated by cross-reference).

A signature field has an associated digital signature value 883, asshown in FIG. 21. Any digital ink captured in a signature field's zoneis automatically verified with respect to the identity of the owner ofthe pen, and a digital signature of the content of the form of which thefield is part is generated and assigned to the field's value. Thedigital signature is generated using the pen user's private signaturekey specific to the application which owns the form. Online signatureverification is well-understood (see, for example, Plamondon, R. and G.Lorette, “Automatic Signature Verification and Writer Identification—TheState of the Art”, Pattern Recognition, Vol. 22, No. 2, 1989, thecontents of which are herein incorporated by cross-reference).

A field element is hidden if its “hidden” attribute is set. A hiddenfield element does not have an input zone on a page and does not acceptinput. It can have an associated field value which is included in theform data when the form containing the field is submitted.

“Editing” commands, such as strike-throughs indicating deletion, canalso be recognized in form fields.

Because the handwriting recognition algorithm works “online” (i.e. withaccess to the dynamics of the pen movement), rather than “offline” (i.e.with access only to a bitmap of pen markings), it can recognize run-ondiscretely-written characters with relatively high accuracy, without awriter-dependent training phase. A writer-dependent model of handwritingis automatically generated over time, however, and can be generatedup-front if necessary,

Digital ink, as already stated, consists of a sequence of strokes. Anystroke which starts in a particular element's zone is appended to thatelement's digital ink stream, ready for interpretation. Any stroke notappended to an object's digital ink stream is appended to the backgroundfield's digital ink stream.

Digital ink captured in the background field is interpreted as aselection gesture. Circumscription of one or more objects is generallyinterpreted as a selection of the circumscribed objects, although theactual interpretation is application-specific.

Table 1 summarises some typical pen interactions with a netpage.

TABLE 1 Summary of some typical pen interactions with a netpage ObjectType Pen input Action Hyperlink General Click Submit action toapplication Form Click Submit form to application Selection Click Submitselection to application Form field Checkbox Any mark Assign true tofield Text Handwriting Convert digital ink to text; assign text to fieldDrawing Digital ink Assign digital ink to field Signature SignatureVerify digital ink signature; generate digital signature of form; assigndigital signature to field None — Circumscription Assign digital ink tocurrent selection

The system maintains a current selection for each pen. The selectionconsists simply of the most recent stroke captured in the backgroundfield. The selection is cleared after an inactivity timeout to ensurepredictable behavior.

The raw digital ink captured in every field is retained on the netpagepage server and is optionally transmitted with the form data when theform is submitted to the application. This allows the application tointerrogate the raw digital ink should it suspect the originalconversion, such as the conversion of handwritten text. This can, forexample, involve human intervention at the application level for formswhich fail certain application-specific consistency checks. As anextension to this, the entire background area of a form can bedesignated as a drawing field. The application can then decide, on thebasis of the presence of digital ink outside the explicit fields of theform, to route the form to a human operator, on the assumption that theuser may have indicated amendments to the filled-in fields outside ofthose fields.

FIG. 22 shows a flowchart of the process of handling pen input relativeto a netpage. The process consists of receiving (at 884) a stroke fromthe pen; identifying (at 885) the page instance 830 to which the page ID50 in the stroke refers; retrieving (at 886) the page description 5;identifying (at 887) a formatted element 839 whose zone 58 the strokeintersects; determining (at 888) whether the formatted elementcorresponds to a field element, and if so appending (at 892) thereceived stroke to the digital ink of the field value 871, interpreting(at 893) the accumulated digital ink of the field, and determining (at894) whether the field is part of a hyperlinked group 866 and if soactivating (at 895) the associated hyperlink; alternatively determining(at 889) whether the formatted element corresponds to a hyperlinkelement and if so activating (at 895) the corresponding hyperlink;alternatively, in the absence of an input field or hyperlink, appending(at 890) the received stroke to the digital ink of the background field833; and copying (at 891) the received stroke to the current selection826 of the current pen, as maintained by the registration server.

FIG. 22A shows a detailed flowchart of step 893 in the process shown inFIG. 22, where the accumulated digital ink of a field is interpretedaccording to the type of the field. The process consists of determining(at 896) whether the field is a checkbox and (at 897) whether thedigital ink represents a checkmark, and if so assigning (at 898) a truevalue to the field value; alternatively determining (at 899) whether thefield is a text field and if so converting (at 900) the digital ink tocomputer text, with the help of the appropriate registration server, andassigning (at 901) the converted computer text to the field value;alternatively determining (at 902) whether the field is a signaturefield and if so verifying (at 903) the digital ink as the signature ofthe pen's owner, with the help of the appropriate registration server,creating (at 904) a digital signature of the contents of thecorresponding form, also with the help of the registration server andusing the pen owner's private signature key relating to thecorresponding application, and assigning (at 905) the digital signatureto the field value.

7 SELECTION OF OBJECT VIA NETPAGES

7.1 Content Selection and Association of Objects

The netpage system provides a mechanism to allow users to select anobject on a printed netpage and submit it to an application, e.g. toassociate the selected object with another object in the netpage system.

In one preferred embodiment, the selection mechanism is effected bycircumscribing the graphical representation of an object using thenetpage pen. The user may then submit the selected object to anapplication by activating a selection hyperlink via the same or anotherprinted netpage.

The registration server maintains a current selection for each pen,describing a region of a page instance from which the selection has beenmade. This description includes the most recent digital ink strokecaptured by way of transmitted signals from the pen relative to thebackground area of the page. Pen strokes are interpreted in anapplication-specific manner once they are submitted to an applicationvia a selection hyperlink activation. When the user ‘clicks’ on aselection hyperlink, the page server obtains the pen's current selectionfrom the registration server and transmits it to the correspondingapplication as part of the selection hyperlink activation, thusassociating the two objects.

When the application receives a selection hyperlink activation, itretrieves the content of the selection from the page server whichmanages the page from which the selection was made. The application maythen retrieve the selection as formatted data, allowing it to interpretthe object in the form of the selected region in an application-specificmanner in relation to the formatted data, or as unformatted data,allowing it to rely on the page server to interpret the selection regionin a meaningful way.

When requesting unformatted data, the application may specify a desiredscope to assist the interpretation of the selected region by the pageserver. Possible scopes include letter, word, phrase, line, paragraphand article. If the page server is unable to interpret the selectionregion according to the desired scope, it may reject the application'srequest for the content of the the selection.

The selection content returned by the page server to the application mayinclude field values. Typically however, only field values which havebeen submitted as part of a form submission are included.

An author of a document can assist selection of articles by grouping allthe elements of an article into an article group 507, as shown in FIG.31. If the application specifies an article as the scope in itsselection request, the page server attempts to find an article grouprelated to the selection region.

The protocol for selection hyperlink activation and subsequent selectioncontent retrieval, illustrated diagrammatically in FIG. 27, operates asfollows.

When page server A receives a selection hyperlink ‘click’, it retrieves510 the current selection associated with the pen from the registrationserver. The selection is described by a page ID 511 and a region 512.The page server then constructs a selection hyperlink request 508 (i.e.a specialization of the hyperlink request 934, shown in FIG. 23). Theselection hyperlink request contains the pen's current selection asshown in FIG. 25. The page server then transmits its own server ID, thehyperlink request ID, and the link ID to the application in the usualway. It also sends the pen's current selection to the application. Theapplication uses the page ID in the current selection to identify thepage server managing the page on which the selection was made (ie. pageserver A). It then requests the content of the selection from this pageserver in the desired format 514 and according to the desired scope 513.Page server B uses the server ID and hyperlink request ID supplied bythe application to obtain the selection from original page server A.Page server B obtains the selection from page server A rather than fromthe application in order to ensure that the application does not modifythe selection to obtain information not intended by the user.

As an alternative, page server A could sign the selection sent to theapplication, allowing page server B to easily verify that the selectionsupplied by the application has not been modified. Once page server Bhas the selection, it retrieves the selected page from its database anddetermines the content of the selection according to the application'sdesired format and scope. Finally the page server returns the selectioncontent 515 to the application for application-specific processing.

If the page on which the selection was made was generated by the sameapplication as that handling the selection hyperlink activation, thenthe application has direct access to the page (i.e. the application canretrieve the entire page from the relevant page server by way of thepage ID of the page) and may already have done so before the selectionhyperlink is activated. In this case, the application preferablyinterprets the selection region without reference to the selectioncontent retrieval mechanism.

Selection of an object may of course alternatively be performed using adraggable command. This has an advantage in that a selection stroke canbe distinguished from a normal input stroke wherever the stroke is made.The user may therefore select an object without inadvertently enteringan input stroke into a field. As such, a user is also able to select anobject that resides entirely within a field relatively easily.

The application may define the selection command in the form of aselection page server command 509, as shown in FIG. 26. This may beplaced in a standard location on all netpages, to provide consistentsupport for selection in much the same way that there is consistentsupport for page duplication.

When a selection page server command is activated by a user, the pageserver forwards the corresponding stroke to the registration server tobe recorded as the current selection for the pen. Apart from thisdifference, the selection mechanism operates in the same way aspreviously described.

As an example, in a netpage e-mail application, clicking on an<attach>button at the bottom of every e-mail composition page effectsattachment of the current selection at the current end of the body ofthe e-mail. The attachment can consist of any object or objects, whichare capable of selection, on any netpage page. The user may have madethis selection from, say, selecting text (eg.word/sentence/paragraph/article) from another netpage page, or selectinga photograph (eg. by circumscription) from a photo collection page. Theentire e-mail may then be reprinted with the attachment included,additional pages being automatically added to the e-mail to accommodatethe attachment.

7.2 Text Selection and Operations Thereon

FIG. 28 shows the styled text object 855 of FIG. 7 decomposed into a setof styled paragraphs 1012, styled words 1014 and styled characters 1016.Styling may be applied at any level. In the absence of styling at aparticular level styling is inherited from the parent level. Each styledcharacter 1016 has an associated character code. This may utilize theUnicode encoding scheme or another encoding scheme.

A formatted element 835 associated with a textflow element 848 is aformatted textflow fragment 1018. The formatted textflow fragmentconsists of a set of formatted text lines 1020. Each formatted text linehas a spatial extent or zone on the page. Each formatted text lineconsists of a set of formatted word fragments 1022, each of which hasits own zone. The zone of formatted text line is the union of the zonesof its word fragments. Each formatted word fragment is associated with astyled word 1014. Where a styled word is broken across multiple lines ithas multiple formatted word fragments. Where the entire styled word lieswithin a single line it has a single formatted word fragment. Eachformatted word fragment consists of a set of formatted characters 1024,each of which has a zone and is associated with a styled character. Thezone of a formatted word fragment is the union of the zones of itsformatted characters.

FIG. 29 shows a fragment 1030 of a printed page bearing a singletwo-line paragraph of text. The words (“in Spain”) are showncircumscribed by a stroke 1032 of a netpage pen.

FIG. 30 shows a fragment 1034 of a printed page including a set ofcommand buttons 1036, 1038, 1040, 1042, 1044 and 1046 designed tooperate on the current selection. It also includes a text field 1048 forentering an arbitrary text string.

FIG. 31 shows part of the object diagram corresponding to the styledtext of FIG. 29. The styled text consists of a single styled paragraph1012 a, the first two styled words 1014 a and 1014 b of which are shown.Because the words are unbroken, they consist of one formatted wordfragment each, 1022 a and 1022 b respectively. The two styled words eachconsists of a set of styled characters 1016 a-c and 1016 d-grespectively. The two formatted word fragments each consists of a set offormatted characters 1022 a-c and 1022 d-g respectively. As shown in theFigure, formatted character 1022 d corresponds to the initial letter “r”of “rain”.

The user can utilize a number of different pen gestures to effect textselection. A click on a word can be used to select that word. Theposition of the click is compared with the zone of each word, and theword whose zone it lies within is selected. Circumscription (orlassoing) can be used to select of one or more words. The degree ofoverlap between the region enclosed by the circumscribing stroke and thezone of each word is used to determine whether the word is selected. Therequired overlap can be configured by the user. Underlining can be usedto select one or more words. A line-like stroke which doesn't otherwiseselect any words is interpreted as an underline. Since the orientationof the text is known, the text which lies above the underlining strokecan be identified.

As described previously, the current selection is available to anapplication which is the target of a selection hyperlink 864. When anapplication receives a selection hyperlink activation, it can requestthe current selection.

A number of useful applications can utilize text selections. Theselected text can be copied to the clipboard of the user's graphicaluser interface (GUI) operating system (e.g. Microsoft Windows, AppleMacintosh OS X, or Linux) for use by other GUI applications. FIG. 30shows a <copy text> command button 1038 which causes the currentlyselected text to be copied to the clipboard. Variants of this functioncan be provided, such as <copy & paste text> which copies and thenpastes the selected text into the currently active GUI application.Under Microsoft Windows this can usually be effected by inserting acontrol-V keystroke into the keystroke input queue. It can also beeffected using Windows Automation, by identifying and invoking thecurrent application's Paste method.

In addition to utilizing an explicit <copy text> command, the user canoperate in a mode where all strokes, if interpretable as selectionstrokes, cause the selected text to be automatically copied to theclipboard of the GUI operating system. The user can also cause suchcopying to occur by utilizing one or more specific gestures, such astapping on a word twice, or circumscribing a set of words twice, orwriting the letter “C” and then circling it.

FIG. 30 shows a <search> command button 1040 which causes the currentlyselected text to be used in a search, such as a search of the World-wideWeb using a search engine such as Google or Yahoo!. Invoking a Websearch engine consists of constructing a URI which identifies the searchengine and includes the selected text as a parameter, and then issuingan HTTP GET or POST using that URI. For example, in the case of Googleit's a matter of appending the selected text to“http://www.google.com/search?q=”, separating multiple words by “+”, andoptionally enclosing multiple words in quotes (“% 22”) to cause them tobe interpreted as a single phrase. The result of the search is eitherdisplayed on the user's desktop or mobile computer, on the user's mobilephone or PDA, or is printed on a netpage printer, all according to theuser's configured preference. The search command may also be linked to asearch of the user's personal file system.

FIG. 30 shows a <dictionary> command button 1042 which causes thecurrently selected text to be used in a dictionary lookup, such as alookup using a Web-based dictionary. The result is either displayed orprinted according to the user's configured preference.

FIG. 30 shows an <encyclopedia> command button 1044 which causes thecurrently selected text to be used in an encyclopedia lookup, such as alookup using a Web-based encyclopedia such as wikipedia. The result iseither displayed or printed according to the user's configuredpreference.

FIG. 30 shows a <translate> command button 1046 which causes thecurrently selected text to be translated into the user's preferred (orpreviously specified) language. The language of selected text is oftenknown from the resource description 842 associated with the styled textobject 855. Where the document 836 is derived from a Webpage, thelanguage is commonly known. The result is either displayed, printed oroutput as sampled or synthesized audio according to the user'sconfigured preference.

Each command button 1038 through 1046 is implemented as a selectionhyperlink.

When user invokes a selection hyperlink which causes the targetapplication to request the currently selected text, it is possible forthe most recent strokes entered by the user to lie in a text field. As aconfigurable option (and as a per-request option) the system deliversthe corresponding recognized text in place of any previously selectedtext.

FIG. 30 shows a text field 1048. This is provided for the convenience ofthe user to allow the user to manually enter text to be processed in thesame way as a text selection, as described in the previous paragraph.

The entire collection of command buttons shown in FIG. 30, as well asthe text field 1048, can be provided for the user's convenience on aseparate “command card”, or can be optionally printed as a “command bar”at the bottom of every page.

As described earlier in relation to the <copy text> command, the usercan also invoke these commands by utilizing a specific gestures, such aswriting a corresponding letter and then circling it, e.g. “C” for<copy>, “S” for <search>, “D” for <dictionary>, “E” for <encyclopedia>,and “T” for <translate>.

7.3 Arbitrary Object Selection and Clippings

An application which is a target of a selection hyperlink can query thecontent of the most recent selection, and can specify the format thatthe selection should be returned in. For example, the application canuse the selection to retrieve text (a word, multiple words, a line,multiple lines, a paragraph, an article, etc.), an object (such as anyof the object types listed in FIG. 15), a page, or the entire document.In the latter case the Netpage system can export the document as aself-contained file, or provide a portable reference to it (with theunique document ID or document instance ID encoded in a URI).

The Netpage system also allows a user to take a virtual clipping from aprinted page. A virtual clipping appears visually as a fragment of theprinted page, bounded by the hand-drawn stroke used to circumscribe theclipping. The clipping is copied to the clipboard of the operatingsystem of the user's display device, e.g. the user's PC, PDA or mobilephone. The user can subsequently paste the clipping into anotherapplication or document, such as a word processing document, an e-mail,or a blog. A clipping may include, by default, any digital ink strokeson the original page.

FIG. 32 shows a printed page with a clipping stroke. FIG. 33 shows thevirtual clipping corresponding to the clipping stroke.

The application may also request a ” clipping”, i.e. graphic pagecontent clipped to the most recent selection stroke. A clippingoptionally remains active, i.e. hyperlinks and/or fields embedded in thepage become part of the clipping, e.g. encoded in HTML (see Section7.4). A clipping includes, by default, any digital ink strokes on theoriginal page.

Using a non-marking Netpage pointer the user can take a clipping using asingle clipping stroke. Using a marking Netpage pen the user canidentify a stroke as a clipping stroke via a subsequent action, such asclicking on a ‘copy’ command on command card or selecting a ‘copy’command from the option menu. FIG. 30 shows a <copy image> commandbutton 1036. It causes the currently selected image or clipping to becopied to the clipboard of the GUI operating system for use by other GUIapplications. The user may, for example, select an image by tapping it,or the user may select a clipping by circumscribing it.

Many other commands which copy objects to the clipboard of the GUIoperating system are possible. It is also possible to provide a generic<copy>command which causes the current selection to be advertised on theclipboard in multiple formats (e.g. plain text, rich text, HTML, image,document, or document reference). The GUI application which retrievesthe selection from the clipboard can then allow the user to select thedesired format. Most GUI operating systems allow a form of “lazy”copying to the clipboard, where the copying application initially onlycopies an object reference to the clipboard, and only copies the objectto the clipboard in its final format once it is notified that anapplication has attempted to retrieve the object from the clipboard.

7.4 Interactive Clippings

An interactive clipping retains the interactivity of the originalprinted page. Any user interaction with the clipping is interpreted inmuch the same way as a corresponding user interaction with the originalprinted page. At a minimum an interactive clipping supports explicit andimplicit hyperlinks. Depending on implementation, it may also supportgestural commands and the context-sensitive option menu.

An interactive clipping provides the user with greater utility than apassive clipping, and allows the publisher of the original page toobtain value from user interactions with the clipping. A printed Netpagepublication is an extension to the publisher's web presence, and eachinteractive clipping further extends this web presence.

7.4.1 Interactive Clipping Design

As shown in FIG. 34, an interactive clipping 310 has an associatedclipping image 312, a clipping input service 314, clipping information316, and an optional citation 318. The clipping image 312 provides thevisual appearance of the clipping. The clipping input service 314interprets user interactions with the clipping. The clipping information316 identifies the printed page instance (i.e. impression) that theclipping was taken from, and the offset between the clipping and thepage. The clipping information 316 allows the clipping input service tointerpret user input received via the clipping against the pagedescription 5 associated with the impression. The citation 318identifies the publication, issue and page number that the clipping wastaken from, if known, and provides a link to the publication's webpresence.

In order to minimize the size of the clipping 310, the clipping image312 is stored separately and may be shared by a number of clippings. Theclipping image 312 may be stored explicitly, or it may be generated fromthe original impression on demand. For the purpose of generating theclipping image 312 from the impression, the clipping information needsto record the entire clipping path rather than just the offset betweenthe clipping and the impression, as well as the time the clipping tookplace to allow digital ink (etc.) added to the impression after theclipping took place to be ignored.

The interactive clipping is represented using an HTML fragmentcontaining a server-side image map (see W3C, Server-Side Image Maps,HTML 4.01 Specification, W3C Recommendation 24 Dec. 1999,http://www.w3.org/TR/html401/struct/objects.html#h-13.6.2):

<A href=“http://www.example.com/cgi-bin/inputService?id=1234”> <IMGsrc=“http://www.example.com/clippings/id1234.gif” ismap></A>

Using HTML allows the interactive clipping to be supported by popularword processors, e-mail programs and blogs, as well as the clipboards ofpopular operating systems.

For illustrative purposes in the example the clipping ID is 1234. Theanchor element identifies the clipping input service, and the idparameter in the input service URI identifies the clipping. The imageelement identifies the clipping image and the fact that it's an imagemap.

A click on the clipping image generates a URI of the following form:

-   http://www.example.com/cgi-bin/inputService?id=1234?25,37

For illustrative purposes in the example the click coordinates are(25,37). Note that the syntax of the parameters in the URI is notstrictly conformant, since HTML does not define a way to combineimage-map ‘query’ parameters with other query parameters. However, theindicated syntax is generally supported.

FIG. 35 shows the interaction between an HTML/HTTP client 330 such as aweb browser, the Netpage clipping image web service 332, the Netpageclipping input web service 334, the Netpage document service 336 (e.g.the page server 10), and a target web service 338. The HTML/HTTP client330 is typically the relay device 601 which receives interaction datafrom the reader 400, and may take the form of a web-enabled laptopcomputer, mobile phone, PDA etc.

When displaying a clipping, the client uses a clipping image URI toretrieve the corresponding clipping image 312 from the Netpage clippingimage web service 332. The image service 332 may in turn rely on theNetpage document service 336 to render the clipping image from theoriginal impression. In this case the image service 332 extracts theclipping ID from the clipping image URI and passes it to the Netpagedocument service 336.

When handling a click on the clipping image 312, the client 330generates a URI for the Netpage clipping input web service 334,including the clipping ID and click location. The input service 334 inturn relies on the Netpage document service 336 to identify theimpression from the clipping ID, and interpret the click locationrelative to the impression (suitably adjusted according to the offsetbetween the impression origin and clipping origin). The Netpage documentservice 336 returns the URI of the final target of the click, and theinput service 334 returns this to the client as a redirect. The clientuses the target URI to retrieve and display the target web page (orother web resource) from the target web service 338.

Although the Netpage image web service 332, input web service 334 anddocument service 336 are shown here as separate services, they may beimplemented as multiple interfaces to a single service (e.g. the pageserver 10).

Server-side image maps provide maximum flexibility, since they allow theexact click location to be interpreted against the original impression'spage description. However, some users object to server-side image mapsbecause they don't reveal a click's target URI. Client-side image mapsmay be used in place of server-side image maps with the addedcomplication that implicit hyperlinks, e.g. associated with images andtext, must be made explicit in the HTML of the interactive clipping.

An impression may bear digital ink and other attached objects (e.g.photos). When the user takes a clipping it can be automatically placedon the clipboard in multiple formats, including variants that includeand exclude digital ink (and other objects). Variants can also beincluded that are passive rather than interactive. The user can then usethe ‘paste special’ function provided in most applications to select anappropriate variant of the clipping.

An image-map only supports clicks. To support more complex gestures,such as swipe to search or access to a contextual option menu, aninteractive clipping can be represented using a script (i.e.widget-style). However, this makes it less compatible with extant HTMLclients, either due to their implementation or their security policies.

Where the citation for the clipping is known, this should be displayedbelow the clipping in the form of an anchor linking to the publication'sweb presence, as shown in FIG. 36.

For non-graphical HTML clients it is important for the user to have away of viewing and interacting with the clipping. This can be providedvia a separate anchor, with a distinct URI, included after the image mapanchor. This is preferable to using the alternate text attribute of theimage-map image element, since that generates an image-map click with alocation of (0,0).

The non-graphical anchor should point to a web page that displays theclipping and allows interactions with it. Ideally this web page willalso support script-based access to the contextual option menu.

FIG. 37 shows the appearance of the interactive clipping in anon-graphical HTML client.

CONCLUSION

The present invention has been described with reference to a preferredembodiment and number of specific alternative embodiments. However, itwill be appreciated by those skilled in the relevant fields that anumber of other embodiments, differing from those specificallydescribed, will also fall within the spirit and scope of the presentinvention. Accordingly, it will be understood that the invention is notintended to be limited to the specific embodiments described in thepresent specification, including documents incorporated bycross-reference as appropriate. The scope of the invention is onlylimited by the attached claims.

1. A method of generating a clipping from a printed substrate, saidsubstrate having a graphic image and a position-coding pattern disposedthereon, said position-coding pattern identifying an identity and aplurality of locations, said method comprising the steps of: receiving,in a computer system, interaction data indicative of the identity and atleast one position of a sensing device relative to the substrate, thesensing device generating the interaction data, when operativelypositioned or moved relative to the substrate, by reading at least someof the position-coding pattern; identifying and retrieving at least partof a page description corresponding to the printed substrate using theidentity, said page description including a visual descriptiondescribing the graphic image and an input description describing one ormore interactive elements; determining, using the page description andthe at least one position of the sensing device, a clipping zone of thesubstrate indicated by the sensing device, said clipping zone containinga part of the graphic image; generating the clipping, said clippingcomprising a clipping identity identifying at least said pagedescription and said clipping zone; and copying the clipping to aclipboard.
 2. The method of claim 1, wherein said clipping identifies aclipping image, said clipping image comprising a clipping visualdescription describing the part of the graphic image contained in saidclipping zone.
 3. The method of claim 2, further comprising the step ofstoring said clipping image on a clipping image store, wherein saidstored clipping image is indexed with said clipping identity.
 4. Themethod of claim 3, further comprising displaying said clipping on adisplay device, said displaying comprising the steps of: retrieving saidclipping image from said clipping image store using said clippingidentity; and displaying said clipping image on the display device. 5.The method of claim 1, further comprising displaying said clipping on adisplay device, said displaying comprising the steps of: identifyingsaid page description using said clipping identity; rendering a clippingimage from said page description, said clipping image comprising aclipping visual description describing the part of the graphic imagecontained in said clipping zone; and displaying said clipping image onthe display device.
 6. The method of claim 1, wherein said clipping zoneis indicated by a path of the sensing device relative to the substrate.7. The method of claim 1, wherein said clipping identity identifies atime at which said clipping is generated.
 8. The method of claim 1,wherein said clipping zone contains at least one interactive element,such that said clipping is an interactive clipping.
 9. The method ofclaim 8, wherein said at least one interactive element is selected fromthe group comprising: a hyperlink and an input field.
 10. The method ofclaim 8, wherein said clipping identity identifies a clipping inputdescription describing said at least one interactive element via saidpage description and said clipping zone.
 11. The method of claim 1,wherein said clipping further identifies a clipping input serviceconfigured for interpreting an interaction with said clipping via agraphical user interface.
 12. The method of claim 1, wherein saidclipping further identifies citation information associated with thepart of the graphic image contained in said clipping zone.
 13. Themethod of claim 1, wherein the clipping is represented, at leastpartially, using an HTML fragment containing an image map.
 14. Themethod of claim 13, wherein the image map is selected from the groupcomprising: a server-side image map and a client-side image map.
 15. Themethod of claim 1, wherein said clipping is represented using a script.16. The method of claim 1, further comprising at least one of: pastingthe clipping into an application, e-mailing the clipping, and pastingthe clipping into a blog.
 17. The method of claim 8, further comprisingthe steps of: displaying said clipping; and interacting with said atleast one interactive element using GUI-based software supported by anoperating system of a display device.
 18. The method of claim 17,wherein the interaction with the at least one interactive elementidentifies a click location and the clipping identity.
 19. The method ofclaim 18, wherein the clipping identity and the click location are usedto identify the corresponding page description and interpret theinteraction.
 20. The method of claim 19, further comprising identifyinga target URI and displaying a corresponding target resource on a displaydevice.